Rate control and video denoising for noisy video data

ABSTRACT

A method and a computer-readable medium containing computer program for simultaneously performing rate control and video denoising for video corrupted by noise are presented. Video data includes original video data and noise data. A variance of the noise data for each macroblock within a frame is estimated. Based on the estimated noise variance, a low bound quantization parameter is generated. The video data are reconstructed such that the distortion between the original video data and reconstructed video data is minimized over each macroblock within a frame. The minimization of the distortion between the original video and reconstructed video data is implemented using a quantization parameter that is equal to or larger than the low bound quantization parameter.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.60/929,992, filed Jul. 20, 2007, which is hereby incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

The present subject matter relates to simultaneously performing ratecontrol and video denoising for video partly or fully corrupted bynoise.

BACKGROUND OF THE INVENTION

Due to the huge size of raw digital video data or image sequences,compression may be applied to such data so that they may be transmittedand stored. Hybrid video coding scheme is the most common video codingscheme. It utilizes motion estimation (ME), discrete cosine transform(DCT)-based transform, and entropy coding to exploit temporal, spatial,and data redundancies, respectively. Most of the existing video codingstandards conform to this hybrid scheme, such as ISO/IEC MPEG-1, MPEG-2,and MPEG-4 standards, and ITU-T H.261, H.263, and H.264 standards.ISO/IEC MPEG-1 and MPEG-2 standards are used extensively by theentertainment industry to distribute movies, in the form of applicationssuch as video compact disks (VCDs) (MPEG-1), digital video or digitalversatile disks (DVDs) (MPEG-2), recordable DVDs (MPEG-2), digital videobroadcast (DVB) (MPEG-2), video-on-demands (VODs) (MPEG-2), highdefinition television (HDTV) in the US (MPEG-2), etc. MPEG-4 is moreadvanced than MPEG-2 and can achieve high quality video at lower bitrate, making it suitable for video streaming over internet, digitalwireless network (e.g. 3G network), multimedia messaging service (MMSstandard from 3GPP), etc. MPEG-4 is accepted for the next generationhigh definition DVD (HD-DVD) standard and the MMS standard. ITU-T H.261,H.263, and H.264 standards are designed mainly for low-delay videophones and video conferencing systems. The earlier H.261 standard wasdesigned to operate at bit rates of p*64 kbit/s, with p=1, 2, . . . ,31. The later H.263 standard is successful and widely used in modernvideo conferencing systems, and video streaming in broadband andwireless networks, including the multimedia messaging service (MMS) in2.5G and 3G networks and beyond. The latest H.264 (also called MPEG-4Ver. 10 or MPEG-4 AVC) is currently the state-of-the-art videocompression standard. It is so powerful that the Moving Picture ExpertsGroup (MPEG) decided to jointly develop the standard with theInternational Telecommunication Union (ITU) TelecommunicationStandardization Sector (ITU-T) in the framework of the Joint Video Team(JVT). The new standard is called H.264 in ITU-T and is called MPEG-4Advance Video Coding (MPEG-4 AVC) or MPEG-4 Version 10 in ISO/IEC. H.264is used in the HD-DVD standard, Direct Video Broadcast (DVB) standard,MMS standard, etc. Based on H.264,.a related standard called the AudioVisual Standard (AVS) is currently under development in China. AVS 1.0is designed for high definition television (HDTV). AVS-M is designed formobile applications. Other related standards may be under development aswell.

H.264 has superior objective and subjective video quality over MPEG-1,MPEG-2, and MPEG-4, and H.261 and H.263. The basic encoding algorithm ofH.264 (Joint Video Team of ITU-T and ISO/IEC JTC 1, “Draft ITU-TRecommendation and Final Draft International Standard of Joint VideoSpecification (ITU-T Rec. H.264 | ISO/IEC 14496-10 AVC),” documentJVT-G050r1, May 2003) is similar to H.263 or MPEG-4 except that integer4×4 DCT is used instead of the traditional 8×8 DCT. There are additionalfeatures of the H.264 standard including intra-prediction mode forl-frames, multiple block sizes and multiple reference frames for motionestimation/compensation, quarter pixel accuracy for motion estimation,in-loop deblocking filter, context adaptive binary arithmetic coding,etc.

The above-mentioned traditional block-based motion compensated videoencoders, e.g. H.263, H.264, are specially designed for the compressionof clean video sequences. However, when the input video sequences arecorrupted by noise, these encoders may have some problems, e.g. they maygenerate bitstreams with high bitrate but poor peak signal-to-noiseratio (PSNR) performance. Moreover, the visual quality of outputreconstructed video sequences may be noisy. There is a need to addressthe rate control and denoising problems for the video sequencespartially or fully corrupted by Gaussian noise as shown in FIGS. 1( a)and 1(b). In this work, a simultaneously rate controlling and videodenoising approach using rate distortion optimization is proposed.

SUMMARY OF THE INVENTION

The present subject matter provides a method for simultaneouslyperforming rate control and video denoising for video corrupted bynoise, comprising the steps of: receiving video data (I^(n)) includingoriginal video data (I) and noise data (n); estimating a variance of thenoise data for each macroblock within a frame; generating a low boundquantization parameter (QP_(lowbound)) based on the estimated noisevariance; and reconstructing the video data such that the distortionbetween the original video data (I) and reconstructed video data (Î^(n))is minimized over each macroblock within a frame.

According to another aspect, the subject matter includes a method forencoding noisy video signals, comprising the steps of: receiving videodata (I^(n)) including original video data (I) and noise data (n);determining a variance of the noise data for each macroblock;determining a low bound quantization parameter (QP_(lowbound)) based onthe noise variance; and controlling the bitrate of output bitstreams andthe quality of reconstructed video data (Î^(n)) based on the low boundquantization parameter.

According to further aspect, the subject matter includes acomputer-readable medium containing a computer program for encodingnoisy video data that include original video data and noise data, saidcomputer program comprising: first code for estimating a variance of thenoise data for each macroblock within a frame; second code forgenerating a low bound quantization parameter based on the estimatednoise variance; and third code for reconstructing the video data suchthat the distortion between the original video data and reconstructedvideo data is minimized over each macroblock within a frame.

In an embodiment of the present method, the minimization of thedistortion between the original video data and reconstructed video datais implemented using a quantization parameter (QP) that is equal to orlarger than the low bound quantization parameter (QP_(lowbound)).

In another embodiment of the present method, the steps of determiningthe low bound quantization parameter and controlling the bitrate ofoutput bitstreams and the quality of reconstructed video data areimplemented by a same component.

According to still further aspect, the subject matter includes a methodfor simultaneously performing rate control and video denoising for videocorrupted by noise, comprising the steps of: receiving video data(I^(n)) including original video data (I) and noise data (n); andreconstructing the video data such that the distortion between theoriginal video data (I) and reconstructed video data (Î^(n)) isminimized by determining the optimal value (R_(t)) for the bitrate ofeach Macroblock.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the subject matter will be apparent withreference to the examples in the following description and withreference to the accompanying drawings, wherein

FIG. 1( a) shows video sequences partially corrupted by noise; and FIG.1( b) shows video sequences fully corrupted by noise.

FIG. 2 is a block diagram showing a video encoder.

FIG. 3 shows rate distortion performance of H.264 for noisy video.

FIG. 4( a) shows the relationship between the correlation D_(i) ^(qn)and the square root of the quantization distortion √{square root over(D_(i) ^(q))} in “foreman” sequence; FIG. 4( b) shows the same in“akiyo” sequence.

FIG. 5 shows the relationship between the square root of the optimalquantization distortion quantization distortion √{square root over(D_(i) ^(q) _(—) _(opt))} and noise variance σ_(n).

FIG. 6 shows rate distortion performance comparison between a methodusing a spatially constant QP and the proposed method.

FIG. 7 shows subjective visual quality comparison: FIG. 7( a) originalframe; FIG. 7( b) noisy frame with spatially varying noise variance;FIG. 7( c) and FIG. 7( d) reconstructed by using constant QP=24 andQP=32, respectively, where the corresponding average (bitrate kb/s, PSNRdB) are (432.67, 33.37) and (97.72, 32.43); FIG. 7( e) and FIG. 7( f)reconstructed using the proposed adaptive QP rate control method, wherethe corresponding average (bitrate kb/s, PSNR dB) are (461.53, 40.51)and (98.04, 35.86).

FIG. 8 shows rate distortion performance comparison between a methodusing H.264 and the proposed method: FIG. 8( a) is for “Foreman”; andFIG. 8( b) is for “Carphone.”

FIG. 9 shows quality comparison for “Foreman”: FIG. 9( a) originalframes; FIG. 9( b) noisy frames with spatially varying noise variance;FIG. 9( c) reconstructed frames using H.264, where the correspondingaverage (bitrate kb/s, PSNR dB) are (301.26, 34.70); and FIG. 9( d)reconstructed frames using the proposed method, where the correspondingaverage (bitrate kb/s, PSNR dB) are (298.42, 41.27).

FIG. 10 shows quality comparison for “Carphone”: FIG. 10( a) originalframes; FIG. 10( b) noisy frames with spatially varying noise variance;FIG. 10( c) reconstructed frames using H.264, where the correspondingaverage (bitrate kb/s, PSNR dB) are (300.27, 32.85); and FIG. 10( d)reconstructed frames using the proposed method, where the correspondingaverage (bitrate kb/s, PSNR dB) are (293.55, 40.48).

DETAILED DESCRIPTION OF THE INVENTION Problem Formulation

The video corrupted by additive Gaussian noise can be formulated asfollowing:I ^(n) =I+n   (1)

where I=[I₁, I₂ . . . I_(k, I) _(k+1) . . . I_(m)]^(T) is the original(ideal) video while I_(k) is the k-th frame, n=[n₁, n₂ . . . n_(k),n_(k+1) . . . n_(m)]^(T) is the additive Gaussian noise, and I^(n)=[hd 1^(n),I₂ ^(n). . . I_(m) ^(n)]^(T) is the noisy observation of the video.Here, each frame has N pixels. In this case, I_(k), n_(k), I_(k) ^(n)are all length-N vectors.

The video denoising problem is designed to find an estimate Î from theoriginal video based on the noisy observation I^(n). From Y. Chen etal., “Maximum a posteriori based (map-based) video denoising via ratedistortion optimization,” in Proc. IEEE International Conference onMultimedia and Expo, 2007, it is known that the denoising problem can beregarded as a rate distortion optimization problem under someappropriate assumptions, which means that it is possible to performvideo denoising through video encoders. However, directly inputting thenoisy video to the video encoder, as shown in FIG. 2, may result ingeneration of bitstreams with high bitrate but poor PSNR performance, asshown FIG. 3. In order to avoid this case, one should carefully choosethe parameter of the video encoder.

In a traditional block-based motion compensated video encoder, therate-distortion optimization model for a macroblock p is defined asfollows:

$\begin{matrix}{{S_{p}^{*} = {{\arg\;{\min\limits_{S_{p}}{D\left( {p,S_{p}} \right)}}} + {{\lambda(Q)}{R_{p}\left( S_{p} \right)}}}}{{\hat{I}}_{p} = {{\hat{I}}_{p}^{n}\left( S_{p}^{*} \right)}}} & (2)\end{matrix}$where Sp=[mode, motion vector, reference index, quantization parameter]denotes the vector of coding decisions for the macroblock p. Amongthese, the most important parameter is a quantization parameter QP sinceusing too small QP will output the bitstream with high bitrate but poorPSNR performance, as shown in FIG. 3. In the following, it will be firstdescribed in detail about how to generate the low bound quantizationparameter QP_(lowbound). A QP which is smaller than QP_(lowbound) willoutput unreasonable bitstreams with a larger bit rate and largerdistortion. Then, it will be described about the proposed algorithm indetail. Finally, it will be described about several experiments todemonstrate the efficiency and effectiveness of the proposed method.

Low Bound Quantization Parameter QPlowbound

For denoising purpose, one way to denoise is to minimize the distortionbetween the original data I and the reconstructed version Î^(n). It isassumed that minimizing over the whole frame is equivalent to minimizingover each marcoblock (MB) within the frame independently, ignoring anydependency among MBs. In the following, the reconstructed distortionE(D_(p)) for MB p is defined as the expectation of the square differencebetween the original pixel I_(i) and the reconstructed one Î_(i) ^(n).

$\begin{matrix}\begin{matrix}{{E\left( D_{p} \right)} = {E\left\lbrack {\sum\limits_{i \in {MB}_{p}}\left( {I_{i} - {\hat{I}}_{i}^{n}} \right)^{2}} \right\rbrack}} \\{= {\sum\limits_{i \in {MB}_{p}}\left\lbrack {E\left( {I_{i} - I_{i}^{n} + I_{i}^{n} - {\hat{I}}_{i}^{n}} \right)}^{2} \right\rbrack}} \\{= {\sum\limits_{i \in {MB}_{p}}\left\lbrack {E\left( {I_{i}^{n} - {\hat{I}}_{i}^{n} - n_{i}} \right)}^{2} \right\rbrack}} \\{= {\sum\limits_{i \in {MB}_{p}}\left\{ {{E\left( {I_{i}^{n} - {\hat{I}}_{i}^{n}} \right)}^{2} - {2 \times {E\left\lbrack {\left( {I_{i}^{n} - {\hat{I}}_{i}^{n}} \right)n_{i}} \right\rbrack}} + {E\left( n_{i} \right)}^{2}} \right\}}} \\{= {\sum\limits_{i \in {MB}_{p}}\left( {D_{i}^{q} - {2 \times D_{i}^{qn}} + \sigma_{n}^{2}} \right)}}\end{matrix} & (3)\end{matrix}$where D_(i) ^(q)=E(I_(i) ^(n)=Î_(i) ^(n))² is the quantized distortion,D_(i) ^(qn)=E[9I_(i) ^(n)−Î_(i) ^(n))n_(i)] is the correlation betweenthe quantized error I_(i) ^(n)−Î_(i) ^(n) and the noise n_(i).

From Equation (3), it can be seen that the expected distortion E(D_(p))is a function of the quantization distortion D_(i) ^(q) and thecorrelation D_(i) ^(qn). In order to minimize E(D_(p)), Applicantsderived the relation between D_(i) ^(q) and D_(i) ^(qn) throughexperimental results. Applicants used the first 50 frames of “foreman”and “akiyo” sequences for training. The relationship between D_(i) ^(q)and D_(i) ^(qn) is shown in FIG. 4. From FIG. 4, D_(i) ^(qn) can beapproximated as a function of √{square root over (D_(i) ^(q))}:

$\begin{matrix}{D_{i}^{qn} = \left\{ \begin{matrix}{{{a_{1}^{n}\left( {\sqrt{D_{i}^{q}} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}},} & {{{for}\mspace{20mu}\sqrt{D_{i}^{q}}} < b^{n}} \\{{{a_{1}^{n}\left( {b^{n} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}},} & {{{for}\mspace{20mu}\sqrt{D_{i}^{q}}} \geq b^{n}}\end{matrix} \right.} & (4)\end{matrix}$where a₁ ^(n), a₂ ^(n), a₃ ^(n), amd b^(n) are parameters determined bythe noise variance. Once the noise variance is fixed, they are constant.Moreover, from FIG. 4, we can see that a₁ ^(n) and b^(n) are bothpositive.

According to the Equation (4), we can re-write E(Dp) as:

$\begin{matrix}{{E\left( D_{p} \right)} = \left\{ \begin{matrix}{{\sum\limits_{i \in {MB}_{p}}\left\{ {D_{i}^{q} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {\sqrt{D_{i}^{q}} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + \sigma_{n}^{2}} \right\}},} & {{{for}\mspace{14mu}\sqrt{D_{i}^{q}}} \leq b^{n}} \\{{\sum\limits_{i \in {MB}_{p}}\left\{ {D_{i}^{q} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {b^{n} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + \sigma_{n}^{2}} \right\}},} & {{{for}\mspace{14mu}\sqrt{D_{i}^{q}}} \geq b^{n}}\end{matrix} \right.} & (5)\end{matrix}$

From the equation above, we can see that E(D_(p)) is a piecewisefunction of √{square root over (D_(i) ^(q))}. In the following, wederive the minimal value of E(D_(p)) for each region.

1) Case one √{square root over (D_(i) ^(q))}≧b^(n):

${E\left( D_{p} \right)} = {\sum\limits_{i \in {MB}_{p}}\left\{ {D_{i}^{q} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {b^{n} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + \sigma_{n}^{2}} \right\}}$is a monotonic increase function of √{square root over (D_(i) ^(q))}when √{square root over (D_(i) ^(q))}≧b^(n)>0. Therefore, the minimalvalue of E(D_(p)) is achieved when √{square root over (D_(i)^(q))}=b^(n). Here, the minimal value is:

$\begin{matrix}{{{{E\left( D_{p} \right)}_{\min} = {\sum\limits_{i \in {MB}_{p}}\left\{ {\left( b^{n} \right)^{2} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {b^{n} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + \sigma_{n}^{2}} \right\}}},{when}}{\sqrt{D_{i}^{q}} = b^{n}}} & (6)\end{matrix}$

2) Case two √{square root over (D_(i) ^(q))}≦b^(n): If √{square rootover (D_(i) ^(q))}≦b^(n),

${E\left( D_{p} \right)} = {\sum\limits_{i \in {MB}_{p}}\left\{ {D_{i}^{q} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {\sqrt{D_{i}^{q}} - {\overset{\_}{a}}_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + \sigma_{n}^{2}} \right\}}$is a second order function of √{square root over (D_(i) ^(q))}. Bytaking the first and second order derivatives over √{square root over(D_(i) ^(q))}, we can get:

$\begin{matrix}{{{\frac{\partial{E\left( D_{p} \right)}}{\partial\sqrt{D_{i}^{q}}} = {{2\sqrt{D_{i}^{q}}} - {2\sigma_{n}2{a_{1}^{n}\left( {\sqrt{D_{i}^{q}} - a_{2}^{n}} \right)}}}},{with}}{{{\sqrt{D_{i}^{q}} \leq {b^{n}.\frac{\partial^{2}{E\left( D_{p} \right)}}{\partial^{2}\sqrt{D_{i}^{q}}}}} = {2 - {4a_{1}^{n}\sigma_{u}}}},{with}}\mspace{14mu}{\sqrt{D_{i}^{q}} \leq {b^{n}.}}} & (7)\end{matrix}$

${{{Let}\mspace{14mu}\frac{\partial{E\left( D_{p} \right)}}{\partial\sqrt{D_{i}^{q}}}} = 0},$we can get:

$\begin{matrix}{\sqrt{D_{i}^{q}} = \frac{2a_{1}^{n}\sigma_{n}a_{2}^{n}}{{2a_{1}^{n}\sigma_{n}} - 1}} & (8)\end{matrix}$

Since 2a_(i) ^(n)σ_(n)

1,

$\frac{\partial^{2}{E\left( D_{p} \right)}}{\partial^{2}\sqrt{D_{i}^{q}}} = {{2 - {4a_{1}^{n}g_{n}}} < 0.}$E(D_(p))achieves its maximal value at

$\sqrt{D_{i}^{q}} = {\frac{2a_{1}^{n}\sigma_{n}a_{2}^{n}}{{2a_{1}^{n}\sigma_{n}} - 1}.}$Also, E(D_(p)) is monotonic increasing when

${0 \leq \sqrt{D_{i}^{q}} \leq \frac{2a_{1}^{n}\sigma_{n}a_{2}^{n}}{{2a_{1}^{n}\sigma_{n}} - 1}},$and monotonic decreasing when

$\frac{2a_{1}^{n}\sigma_{n}a_{2}^{n}}{{2a_{1}^{n}\sigma_{n}} - 1} \leq \sqrt{D_{i}^{q}} \leq {b^{n}.}$Therefore, the minimal value of E(D_(p)) is achieved at either √{squareroot over (D_(i) ^(q))}=0 or √{square root over (D_(i) ^(q))}=b^(n).

$\begin{matrix}{\left. {E\left( D_{p} \right)} \right|_{\sqrt{D_{i}^{q}} = 0} = {\left. {0 - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {0 - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + {\sigma_{n}^{2}.{E\left( D_{p} \right)}}} \right|_{\sqrt{D_{i}^{q}} = b^{n}} = {\left( b^{n} \right)^{2} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {b^{n} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + {\sigma_{n}^{2}.}}}} & (9)\end{matrix}$

According to FIG. 4, we can see that 2a_(i) ^(n)σ_(n)

1 and

$b^{n} > {2\frac{2\sigma_{n}a_{1}^{n}}{{2\sigma_{n}a_{1}^{n}} - 1}{a_{2}^{n}.}}$Thus,

$\left\lbrack {{E\left( D_{p} \right)}❘_{\sqrt{D_{i}^{4}} = b^{n}}{{- {E\left( D_{p} \right)}}❘_{\sqrt{D_{i}^{4}} = 0}}} \right\rbrack = {{\left( b^{n} \right)^{2} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {b^{n} - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + \sigma_{n}^{2} - \left\{ {0 - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {0 - a_{2}^{n}} \right)}^{2} + a_{3}^{n}} \right\rbrack}} + \sigma_{n}^{2}} \right\}} = {{b^{n}\left\lbrack {{b^{n}\left( {1 - {2\sigma_{n}a_{1}^{n}}} \right)} + {4\sigma_{n}a_{1}^{n}a_{2}^{n}}} \right\rbrack} < {b^{n}\left\lbrack {{2\frac{2\sigma_{n}a_{1}^{n}}{{2\sigma_{n}a_{1}^{n}} - 1}{a_{2}^{n}\left( {1 - {2\sigma_{n}a_{1}^{n}}} \right)}} + {4\sigma_{n}a_{1}^{n}a_{2}^{n}}} \right\rbrack} < 0}}$

Therefore, the minimal value of E(Dp) is achieved when √{square rootover (D_(i) ^(q))}=b^(n). Here, the minimal value is:

$\begin{matrix}\begin{matrix}{{E\left( D_{p} \right)}_{\min} = {\sum\limits_{i \in {MB}_{p}}\;\left\{ {\left( b^{n} \right)^{2} - {2{\sigma_{n}\left\lbrack {{a_{1}^{n}\left( {b^{n} - a_{2}^{n}} \right)} + a_{3}^{n}} \right\rbrack}} +} \right.}} \\{\left. \sigma_{n}^{2} \right\},{{{when}\mspace{14mu}\sqrt{D_{i}^{q}}} = b^{n}}}\end{matrix} & (10)\end{matrix}$

According to Equations (6) and (10), we can see that the minimal valueof E(D_(p)) is achieved when √{square root over (D_(i) ^(q))}=b^(n). Letus denote this optimal √{square root over (D_(i) ^(q))} as √{square rootover (D_(i) ^(q) _(—) _(opt))}. As shown in FIG. 4, b^(n) is determinedby σ_(n). Therefore, the optimal √{square root over (D_(i) ^(q) _(—)_(opt))}, which minimizes E(D_(p)), is determined by σ_(n). FIG. 5 showsthe relationship between the optimal √{square root over (D_(i) ^(q) _(—)_(opt))} and σ_(n). We can see that the optimal √{square root over(D_(i) ^(q) _(—) _(opt))} can be approximated as a linear function ofσ_(n):√{square root over (D _(i) ^(q) _(—) _(opt))}=θσ_(n)+ε   (11)

In sufficiently high bit-rate situation, a reasonably well-behavedsource probability distribution can be approximated as a constant withineach quantization interval. This would lead to the typical bit-rateapproximation (T. Wiegand et al., “Rate-constrained coder control andcomparison of video coding standards,” IEEE Trans. Circuits Syst. VideoTechnol., pp. 688-703, 2003):

$\begin{matrix}{D^{q} = \left\{ \begin{matrix}{{\frac{{QP}^{2}}{3},{{for}\mspace{14mu} H{.263}}}\mspace{11mu}} & \; \\{\frac{2^{\frac{{QP} - 12}{3}}}{3},{{for}\mspace{14mu} H{.264}}} & \mspace{14mu}\end{matrix} \right.} & (12)\end{matrix}$

Applicants relaxed the above equation into a more general approximationby introducing a parameter ω and the bit-rate approximation becomes:

$\begin{matrix}{D^{q} = \left\{ \begin{matrix}{\frac{{QP}^{2}}{\omega},{{for}\mspace{14mu} H{.263}}} & \mspace{14mu} \\{\frac{2^{\frac{{QP} - 12}{3}}}{\omega},{{for}\mspace{14mu} H{.264}}} & \mspace{14mu}\end{matrix} \right.} & (13)\end{matrix}$

According to Equations (11) and (13), QP₀ corresponding to √{square rootover (D_(i) ^(q) _(—) _(opt))} can be generated as:

$\begin{matrix}{{QP}_{0} = \left\{ \begin{matrix}{\sqrt{{\omega\left( {{\theta\;\sigma_{n}} + ɛ} \right)}^{2}},{{for}\mspace{14mu} H{.263}}} & \; \\{{{3\mspace{11mu}{\log_{2}\left\lbrack {\omega\left( {{\theta\;\sigma_{n}} + ɛ} \right)}^{2} \right\rbrack}} + 12},{{for}\mspace{14mu} H{.264}}} & \;\end{matrix} \right.} & (14)\end{matrix}$where

QP₀ is an initial quantization parameter;

σ_(n) is the standard deviation of noise variance;

ω,θ and ε are introduced parameters; and

H.263, and H.264 correspond to ITU-T H.263, and H.264 standards.

$\begin{matrix}{{{R^{*}(i)} = {\min\left\{ {{R\left\lbrack {Q\;{P_{lowbound}(i)}} \right\rbrack},{\sqrt{\frac{a}{\lambda^{*}}}{{MAD}^{2}\left( {i,t} \right)}}} \right\}}}{where}{{\sum\limits_{({i = 1})}^{N_{MB}}{R^{*}(i)}} = {R_{t}.}}} & (19)\end{matrix}$

$\begin{matrix}{{\overset{\sim}{R}(i)} = {\min\left( {{R\left\lbrack {Q\;{P_{lowbound}(i)}} \right\rbrack},\frac{{R_{t}(i)}{{MAD}^{2}\left( {i,t} \right)}}{\sum\limits_{k = i}^{N_{MB}}{{MAD}^{2}\left( {k,t} \right)}}} \right)}} & (20)\end{matrix}$From previous analysis, it is known that QP₀ is the quantizationparameter corresponding to √{square root over (D_(i) ^(q) _(—) _(opt))}that minimizes the expectation of the square difference E(D_(p)) betweenthe original pixel l_(i) and the reconstructed one Î_(i) ^(n). In thisprocess, the role of bit rate is not involved. Therefore, this QP₀ as alow bound quantization parameter is treated. Any reasonable QP should beequal to or larger than QP₀ with a smaller bit rate. A QP smaller thanQP₀ will have a larger bit rate and larger distortion, which is notreasonable. Here, the low bound quantization parameter is denoted asQP_(lowbound). Also, QP_(lowbound) can be defined as follows:

$\begin{matrix}{{QP}_{lowbound} = {{QP}_{0} = \left\{ \begin{matrix}{\sqrt{{\omega\left( {{\theta\;\sigma_{n}} + ɛ} \right)}^{2}},{{for}\mspace{14mu} H{.263}}} & \; \\{{{3\mspace{11mu}{\log_{2}\left\lbrack {\omega\left( {{\theta\;\sigma_{n}} + ɛ} \right)}^{2} \right\rbrack}} + 12},{{for}\mspace{14mu} H{.264}}} & \;\end{matrix} \right.}} & (15)\end{matrix}$

The Proposed Simultaneously Rate Control and Video Denoising Algorithm

According to the above analysis, for different σ_(n) ² noise, there is adifferent low bound quantization parameter QP_(lowbound). Therelationship between QP_(lowbound) and σ_(n) is shown in Equation (15).Intuitively, for noisy video that has spatially varying noise variance,we should generate spatially varying QP_(lowbound) to achieve a bestrate-distortion performance. Based on this intuition, Applicants proposetwo rate control schemes to simultaneously denoise and compress thenoisy video that has spatially varying noise variance: one is exhaustivesearch based rate control scheme and the other is optimization basedrate control scheme.

1. Exhaustive Search Based Control Scheme:

The detailed procedures of the exhaustive search based rate controlscheme are described as follows.

1) First, estimating a noise variance for each macroblock.

2) Then, based on the estimated noise variance, computing aQP_(lowbound) by Equation (15) and using it as an initial QP for eachmacroblock. By doing this, an initial QP map, which will be used in thefurther rate control, is created for each frame.

3) Based on the QP map, encoding the frame and generating the actualbitrate R. If the bitrate is not larger than the target bitrateR≦R_(target), the bitstreams are outputted. If R>R_(target), thesmallest QP_(i) in the QP map are increased and returned to step 3).

2. Optimization Based Rate Control Scheme:

For the MB level rate allocation problem, minimization of the overalldistortion of the current frame under the constraint that the overallbitrate is equal to R_(t) is desired. Also, the traditional MB levelrate allocation problem for the noise-free video can be formulated asfollows:

$\begin{matrix}{{\begin{matrix}\min \\{R(i)}\end{matrix}{\sum\limits_{i = 1}^{N_{MB}}\;{D\left( {R(i)} \right)}}},{{s.t.{\sum\limits_{i = 1}^{N_{MB}}\;{R(i)}}} = R_{t}}} & (16)\end{matrix}$where D(R(i)) is the distortion of the i^(th) MB given that the rate ofthe i^(th) MB is R(i).

According to the previous discussion, in order to perform videode-noising using rate distortion optimization, the quantizationparameter QP should be set to be larger than QP_(lowbound). Therefore,we can integrate the de-noising problem and rate control problem bysolving the following optimization problem:

$\begin{matrix}{{\begin{matrix}\min \\{R(i)}\end{matrix}{\sum\limits_{i = 1}^{N_{MB}}\;{D\left( {R(i)} \right)}}},{{{s.t.{\sum\limits_{i = 1}^{N_{MB}}\;{R(i)}}} = R_{t}};{{R(i)} \leq {{R\left( {{QP}_{lowbound}(i)} \right)}.}}}} & (17)\end{matrix}$

For any reasonable rate distortion relationship, the correspondingsolution for the rate allocation problem can be found by solving theoptimization problem above. Here, the rate distortion relationship inH.264 is used as an example. As shown in Y. Chen and O. Au,“Simultaneous RD-Optimized rate control and video de-noising,” in IEEEInt. Conf. Acoustics, Speech, Signal Processing, 2008, the MB level ratedistortion relationship in H.264 can be approximated as

${D(i)} = {\alpha{\frac{{MAD}^{4}\left( {i,t} \right)}{R(i)}.}}$Therefore, the optimization problem becomes:

$\begin{matrix}{{{\begin{matrix}\min \\{R(i)}\end{matrix}{\sum\limits_{i = 1}^{N_{MB}}{a\frac{{MAD}^{4}\left( {i,t} \right)}{R(i)}\mspace{14mu}{s.t.{\sum\limits_{i = 1}^{N_{MB}}\;{R(i)}}}}}} = R_{t}};{{R(i)} \leq {{R\left( {{QP}_{lowbound}(i)} \right)}.}}} & (18)\end{matrix}$

Obviously, the optimization problem above is a convex optimizationproblem. The optimal solution can be generated by solving the KKTconditions:

$\begin{matrix}{{{R^{*}(i)} = \left. \min 〚{\left\lbrack {{R\left( 〛 \right.}{{QP}_{lowbound}(i)}} \right),{\sqrt{\frac{\alpha}{\lambda^{*}}}{{MAD}^{2}\left( {i,t} \right)}}} \right\rbrack}{where}{{\sum\limits_{i = 1}^{N_{MB}}{R^{*}(i)}} = {R_{t}.}}} & (19)\end{matrix}$

The optimal solution can be found through an iterative form of findingλ*. However, since it is too complex, a suboptimal solution is usedinstead. An approximate Lagrangian parameter

${\overset{\_}{\lambda} = \left( \frac{\sqrt{a}{\sum\limits_{i = 1}^{N_{MB}}{{MAD}^{2}\left( {i,t} \right)}}}{R_{t}} \right)^{2}},$which is actually the optimal Lagrangian parameter of the optimizationproblem in equation (16) with

${{D(i)} = {\alpha\frac{{MAD}^{4}\left( {i,t} \right)}{R(i)}}},$is used. Then, the suboptimal solution {tilde over (R)}(i) is:R ⁻(i)=min [([R(QP ₁lowbound (i), (R ₁ t(i)]MAD ¹2(i,t))/(Σ₁(k=i)^(t)(N₁ MB)≡[(MAD ¹ ₂(k,t))])])]  (20)

Experimental Results

The proposed simultaneous rate control and video de-noising algorithm isevaluated based on modified H.264 JM 9.0. In order to simulate thenoising process, the clean video sequences are first manually distortedby adding Gaussian noise. We assume that one slice contains one row MBsand each slice has a possibility p to be corrupted by Gaussian noise andpossibility 1-p to be error-free. Then, the noisy videos are input tothe video encoder. The video sequences “Foreman” and “Carphone” in QCIFformat are tested. In all experiments as conducted, parameters ω, θ, andε are set to be 1.694, 1.049, and 0.445, respectively (Y. Chen, “VideoDenoising and Error Concealment for Robust Video Communication,” inMaster Thesis, 2007). In the following, the results of σ_(n) ²=100 andp=0.4 will be shown. Similar results are obtained using different σ_(n)² and p. The PSNR is computed by comparing with the original videosequence.

In the first experiment, the proposed exhaustive search based ratecontrol scheme is compared with H.264 encoder using spatially constantQP. The rate distortion performance is shown in FIG. 6. It can be seenthat the proposed method has much better performance and can gain up to7 dB, compared with the method using a spatially constant QP. Thesubjective visual quality is also examined in FIG. 7, where FIGS. 7( a)and 7(b) are the original and noisy frames. FIGS. 7( c) and 7(d) are theresults reconstructed by using spatially constant QP=24 and QP=32,respectively, while FIGS. 7( e) and 7(f) are the results generated usingthe current spatially adaptive QP method. Applicants used QP=35 for thenoisy slices for both FIGS. 7( e) and 7(f). For error-free slices,Applicants used QP=12 and QP=26 for FIGS. 7( e) and 7(f), respectively.The corresponding (bitrates (kb/s), PSNR (dB)) for FIGS. 7( c)-7(f) are(432.67, 33.37), (97.72, 32.43), (461.53, 40.51), and (98.04, 35.86),respectively. It can be seen that, by using the proposed spatiallyadaptive QP rate control method, not only the noise in the noisy slicesis greatly reduced, but also the detail information in the error-freeslices can be well-preserved. On the contrary, using spatially constantQP method will not only have a poor denoising performance for the noisyslices but also introduce serious distortion to the error-free slices.Notice that the artifacts not only come from the compression distortionof current frame but also from the distortion propagation of previousframes.

In the second experiment, the proposed optimization based rate controlscheme is compared with the rate control method in H.264 (Z. G. Li, F.Pan, K. P. Lim, and S. Rahardja, “Adaptive Rate Control for H.264,”inProc. IEEE International Conference on Image Processing, 2004). The ratedistortion performance is shown in FIG. 8. It can been seen that thePSNR performance of the proposed method is much better than that of themethod in H.264, especially at high bit-rate situation. Up to 7.88 dBgain can be obtained. The subjective visual quality is also examined inFIGS. 9 and 10, where FIGS. 9( a) and 10(a) and FIGS. 9( b) and 10(b)are the original and noisy frames, respectively. FIGS. 9( c) and 10(c)are the results reconstructed by the rate control method in H.264 whileFIGS. 9( d) and 10(d) are the results generated using the proposedmethod at bit-rate about 300 kb/s. It can be seen that by using theproposed rate control method, not only the noise in the noisy slices isgreatly reduced, but also the detail information in the error-freeslices can be well-preserved. On the contrary, using the rate controlmethod in H.264 will not only have a poor denoising performance for thenoisy slices but also introduce serious distortion to the error-freeslices. Notice that the artifacts not only come from the compressiondistortion of current frame but also from the distortion propagation ofprevious frames.

The present subject matter can be used in all practical video processingsystems. Particularly, all video encoder/transcoder (e.g., H.263, H.264,MPEG-4, AVS, etc) that require regulating the output rate can use thepresent subject matter. One can incorporate this subject matter intohis/her products to improve the coding efficiency and subjective qualityof video sequences. In addition, when the noise variance is large, onemay integrate other pre-filtering-based denoising method for highbitrate situation.

Although the subject matter has been described with reference to theillustrated embodiment, the subject matter is not limited thereto. Thesubject matter being thus described, it will be obvious that the samemay be varied in many ways. Such variations are not to be regarded as adeparture from the spirit and scope of the subject matter, and all suchmodifications as would be obvious to one skilled in the art are intendedto be included within the scope of the following claims.

What is claimed is:
 1. A method for simultaneously performing ratecontrol and video denoising for video corrupted by noise, comprising thesteps of: receiving video data (I^(n)) including original video data (I)and noise data (n); estimating a variance of the noise data for eachmacroblock within a frame; generating a low bound quantization parameter(QP_(lowbound)) based on the estimated noise variance; and using the lowbound quantization parameter (QP_(lowbound)) to reconstruct the videodata such that distortion between the original video data (I) andreconstructed video data (Î^(n)) is minimized over each macroblockwithin a frame and using the low bound quantization parameter(QP_(lowbound)) to regulate data output rate simultaneously with theminimizing of the distortion.
 2. The method of claim 1, wherein theminimization of the distortion between the original video data andreconstructed video data is implemented using a quantization parameter(QP) that is equal to or larger than the low bound quantizationparameter (QP_(lowbound)).
 3. The method of claim 1, wherein the lowbound quantization parameter is calculated as a function of a quantizeddistortion (Dp).
 4. The method of claim 3, wherein the distortionbetween the original video data and reconstructed video data isrepresented by an expected distortion (E(Dp)) that is a function of thequantized distortion.
 5. The method of claim 4, wherein the expecteddistortion is minimized when the square root of the quantized distortionis equal to a parameter (b^(n)) determined by the noise variance.
 6. Amethod for simultaneously performing rate control and video denoisingfor video corrupted by noise, comprising the steps of: receiving videodata (I^(n)) including original video data (I) and noise data (n);estimating a variance of the noise data for each macroblock within aframe; generating a low bound quantization parameter (QP_(lowbound))based on the estimated noise variance; and using the low boundquantization parameter (QP_(lowbound)) to reconstruct the video datasuch that the distortion between the original video data (I) andreconstructed video data (Î^(n)) is minimized over each macroblockwithin a frame, wherein the low bound quantization parameter iscalculated as a function of a quantized distortion (Dp), wherein thedistortion between the original video data and reconstructed video datais represented by an expected distortion (E(Dp)) that is a function ofthe quantized distortion, wherein the expected distortion is minimizedwhen the square root of the quantized distortion is equal to a parameter(b^(n)) determined by the noise variance, and wherein the low boundquantization parameter (QP_(lowbound)) is calculated by the followingequation: ${QP}_{lowbound} = \left\{ \begin{matrix}{\sqrt{{\omega\left( {{\theta\;\sigma_{n}} + ɛ} \right)}^{2}},{{for}\mspace{14mu} H{.263}}} & \; \\{{{3\mspace{11mu}{\log_{2}\left\lbrack {\omega\left( {{\theta\;\sigma_{n}} + ɛ} \right)}^{2} \right\rbrack}} + 12},{{for}\mspace{14mu} H{.264}}} & \;\end{matrix} \right.$ where QP₀ is an initial quantization parameter;σ_(n) is the standard deviation of noise variance; ω,θ and ε areintroduced parameters; and H.263, and H.264 correspond to ITU-T H.263,and H.264 standards.
 7. The method of claim 2, wherein the bitstreams ofthe reconstructed video data are outputted, if the actual bitrate (R) isnot larger than a predetermined bitrate (R_(target)).
 8. The method ofclaim 7, wherein the reconstruction of the video data is implementedbased on a QP map that includes a QP_(i), corresponding ith macroblockwithin the frame.
 9. The method of claim 8, wherein the smallest QP_(i),in the QP map is increased, if the actual bitrate is larger than thepredetermined bitrate.
 10. The method of claim 1, further comprising:controlling the data output rate of reconstructed video data(Î^(n))based on the low bound quantization parameter.
 11. The method ofclaim 10, wherein the steps of determining the low bound quantizationparameter and controlling the data output rate of reconstructed videodata are implemented by a same component.
 12. A non-transitorycomputer-readable medium containing a computer program for encodingnoisy video data that include original video data and noise data, saidcomputer program comprising: first code for estimating a variance of thenoise data for each macroblock within a frame; second code forgenerating a low bound quantization parameter (QP_(lowbound)) based onthe estimated noise variance; and third code for reconstructing thevideo data by using the low bound quantization parameter (QP_(lowbound))such that distortion between the original video data and reconstructedvideo data is minimized over each macroblock within a frame and forusing the low bound quantization parameter (QP_(lowbound)) to regulatedata output rate simultaneously with the minimizing of the distortion.13. The computer-readable medium of claim 12, wherein the minimizationof the distortion between the original video data and reconstructedvideo data is implemented using a quantization parameter that is equalto or larger than the low bound quantization parameter.
 14. The methodof claim 1, further comprising: reconstructing the video data such thatthe distortion between the original video data (I) and reconstructedvideo data (Î^(n)) is minimized by determining an optimal value (R_(t))for a bitrate of each macroblock.
 15. The method of claim 14, whereinthe optimal value for the bitrate of each macroblock is generated bysolving the following equation:${R^{*}(i)} = {\min\left\{ {{R\left\lbrack {{QP}_{lowbound}(i)} \right\rbrack},{\sqrt{\frac{a}{\lambda^{*}}}{{MAD}^{2}\left( {i,t} \right)}}} \right\}}$where ${{\sum\limits_{({i = 1})}^{N_{MB}}{R^{*}(i)}} = R_{t}},$ R*(i) isthe optimal rate of the ith macroblock, QP_(lowbound) (i) is the lowbound quantization step of the ith macroblock R[QP_(lowbound) (i)] isthe estimated rate of the ith macroblock when the low bound quantizationstep is used a is a constant parameter λ* is the optimal Lagrangianparameter MAD(i,t) is the predicted mean absolution difference of thei-the macroblock at frame t N_(MB) is the total number of macroblocksR_(t) is the target bit rate of frame t.
 16. The method of claim 14,wherein the optimal value for the bitrate of each macroblock isapproximated by the following equation:${{\overset{\sim}{R}(i)} = {\min\left( {{R\left\lbrack {Q\;{P_{lowbound}(i)}} \right\rbrack},\frac{{R_{t}(i)}{{MAD}^{2}\left( {i,t} \right)}}{\sum\limits_{k = i}^{N_{MB}}{{MAD}^{2}\left( {k,t} \right)}}} \right)}},$where {tilde over (R)}(i) is the suboptimal rate of the ith macroblock,QP_(lowbound) (i) is the low bound quantization step of the ithmacroblock R[QP_(lowbound) (i)] is the estimated rate of the ithmacroblock when the low bound quantization step is used MAD(i,t) is thepredicted mean absolution difference of the i-the macroblock at frame tN_(MB) is the total number of macroblocks R_(t)(i) is the number ofremain bits for the remaining MBs in the frame t.